Prompt-tunning Hierarchical Network for Few-shot Multimodal Relation Extraction

doi:10.19678/j.issn.1000-3428.00 70493

Abstract

Abstract: Multimodal Relation Extraction (MRE) method assists relation extraction tasks by utilizing multimodal information. To achieve good relation extraction performance, existing MRE models usually need to be effectively trained on a large amount of labeled data. However, they perform poorly in the case of few-shot samples. To address this issue, this paper fully utilizes the rich semantic and prior knowledge in relation labels and proposes a new prompt-tunning hierarchical network. Firstly, design and implement a text prompt module based on knowledge injection. Specifically, by utilizing the entity information hidden in relation labels and introducing virtual entity type words to construct prompt templates, the model can perceive the potential range of entity types in the sample, and continuously optimize the introduced virtual relation answer words using context to express the best semantic information, thereby improving the performance of the model in the case of few-shot samples. Secondly, by utilizing the mutual constraint relation between entity pairs and relations, design and implement an entity-relations collaborative optimization module to further improve the effect of relation extraction. Finally, in each self-attention layer of the text encoder, a visual prefix based attention mechanism is introduced to deeply integrate the layered multi-scale visual features with the text information, thereby generating more effective and robust text representations, significantly reducing the model's sensitivity to errors. The experimental results on the multimodal neural relation extraction dataset (MNRE) show that the precision, recall, and F1 score of the model reach 84.97%, 83.91%, and 84.43%, respectively, which are all improved compared to the benchmark model. Especially in the case of few-shot samples, the proposed model in this paper is significantly better than the benchmark model, demonstrating good relation extraction performance.

摘要： 多模态关系抽取(MRE)方法通过利用多模态信息辅助关系抽取任务。现有MRE模型要达到良好的关系抽取效果，通常需要在大量标注数据的基础上进行有效训练，然而在少样本情况下却表现不佳，针对上述问题，本文充分利用关系标签中丰富的语义知识和先验知识，提出一种新的提示调优分层网络。首先，设计实现基于知识注入的文本提示模块，具体来说，通过利用关系标签中隐含的实体信息，引入虚拟实体类型词构建提示模板，使模型可以感知样本中潜在的实体类型范围，并利用上下文对引入的虚拟关系答案词进行不断优化，以表达最佳的语义信息，从而提升模型在少样本情况下的性能。其次，利用实体对与关系之间的相互制约关系，设计实现实体-关系协同优化模块，以进一步提升关系抽取效果。最后，在文本编码器的每个自注意力层中，通过引入基于视觉前缀的注意力机制，将分层的多尺度视觉特征与文本信息深度融合，从而生成更加有效且鲁棒的文本表示，显著降低模型对错误的敏感性。在多模态神经关系抽取数据集(MNRE)上的实验结果表明，该模型的精确率，召回率，F1值分别达到84.97%、83.91%与84.43%，较基准模型均有一定提升，特别是在少样本情况下，本文所提模型显著优于基准模型，展现出良好的关系抽取效果。

LI Weigan, GUI Ning, QIAN Yurong, CHEN Jiaying. Prompt-tunning Hierarchical Network for Few-shot Multimodal Relation Extraction[J]. Computer Engineering, doi: 10.19678/j.issn.1000-3428.00 70493.

李维淦, 桂宁, 钱育蓉, 陈嘉颖. 提示调优分层网络用于少样本多模态关系抽取[J]. 计算机工程, doi: 10.19678/j.issn.1000-3428.00 70493.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.00 70493

References

[1] ZHENG C, WU Z, FENG J, et al. MNRE: A Challenge Multimodal Dataset for Neural Relation Extraction with Visual Evidence in Social Media Posts[C]//2021 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2021: 1-6.
[2] 孙紫阳, 顾君忠, 杨静. 基于深度学习的中文实体关系抽取方法[J]. 计算机工程, 2018, 44(9): 164-170. SUN ZIYANG, GU JUNZHONG, YANG JING. Chinese Entity Relation Extraction Method Based on DeepLearning[J]. Computer Engineering, 2018, 44(9): 164-170.
[3] 李冬梅, 张扬, 李东远, 等. 实体关系抽取方法研究综述[J]. 计算机研究与发展, 2020, 57(07): 1424-1448. LI D, ZHANG Y, L D, et al. Review of Entity Relation Extraction Methods [J]. Review of Entity Relation Extraction Methods, 2020, 57(07): 1424-1448.
[4] MOONEY R. Relational Learning of Pattern-Match Rules for Information Extraction[C]//Proceedings of the sixteenth national conference on artificial intelligence. 1999, 328: 334.
[5] HUFFMAN S B. Learning Information Extraction Patterns from Examples[C]//International Joint Conference on Artificial Intelligence. Berlin, Heidelberg: Springer Berlin Heidelberg, 1995: 246-260.
[6] ZHANG Y, ZHONG V, CHEN D, et al. Position-aware Attention and Supervised Data Improve Slot Filling[C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017: 35-45.
[7] GUO Z, NAN G, LU W, et al. Learning Latent Forests for Medical Relation Extraction[C]//Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence. 2021: 3651-3657.
[8] KENTON J D M W C, TOUTANOVA L K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[C]//Proceedings of naacL-HLT. 2019, 1: 2.
[9] LI J, WANG R, ZHANG N, et al. Logic-guided Semantic Representation Learning for Zero-Shot Relation Classification[C]//Proceedings of the 28th International Conference on Computational Linguistics. 2020: 2967-2978.
[10] ZENG D, LIU K, CHEN Y, et al. Distant Supervision for Relation Extraction Via Piecewise Convolutional Neural Networks[C]//Proceedings of the 2015 conference on empirical methods in natural language processing. 2015: 1753-1762.
[11] SOARES L B, FITZGERALD N, LING J, et al. Matching the Blanks: Distributional Similarity for Relation Learning[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019: 2895-2905.
[12] ZHENG C, FENG J, FU Z, et al. Multimodal Relation Extraction with Efficient Graph Alignment[C]//Proceedings of the 29th ACM International Conference on Multimedia. 2021: 5298-5306.
[13] ZHENG C, FENG J, FU Z, et al. Multimodal Relation Extraction with Efficient Graph Alignment[C]//Proceedings of the 29th ACM International Conference on Multimedia. 2021: 5298-5306.
[14] CHEN X, ZHANG N, LI L, et al. Good Visual Guidance Make A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction[C]//Findings of the Association for Computational Linguistics: NAACL 2022. 2022: 1607-1618.
[15] 吴海鹏, 钱育蓉, 冷洪勇. 基于双向注意力机制的多模态关系抽取[J]. 计算机工程, 2024, 50(4): 160-167. WU HAIPENG, QIAN YURONG, LENG HONGYONG. Multimodal Relation Extraction Based on Bidirectional Attention Mechanism[J]. Computer Engineering, 2024, 50(4): 160-167.
[16] XU B, HUANG S, DU M, et al. A Unified Visual Prompt Tuning Framework with Mixture-of-Experts for Multimodal Information Extraction[C]//International Conference on Database Systems for Advanced Applications. Cham: Springer Nature Switzerland, 2023: 544-554.
[17] CUI S, CAO J, CONG X, et al. Enhancing Multimodal Entity and Relation Extraction With Variational Information Bottleneck[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024, 32: 1274-1285.
[18] BROWN T B, MANN B, RYDER N, et al. Language Models are few-shot learners[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020: 1877-1901.
[19] BEN-DAVID E, OVED N, REICHART R. PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains[J]. Transactions of the Association for Computational Linguistics, 2022, 10: 414-433.
[20] HU S, DING N, WANG H, et al. Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022: 2225-2240.
[21] DING N, CHEN Y, HAN X, et al. Prompt-learning forFine-grained Entity Typing[C]//Findings of the Association for Computational Linguistics: EMNLP 2022. 2022: 6888-6901.
[22] GAO T, FISCH A, CHEN D. Making Pre-trained Language Models Better Few-shot Learners[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021: 3816-3830.
[23] SCHICK T, SCHMID H, SCHÜTZE H. Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification[C]//Proceedings of the 28th International Conference on Computational Linguistics. 2020: 5569-5578.
[24] SHIN T, RAZEGHI Y, LOGAN IV R L, et al. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020: 4222-4235.
[25] HAMBARDZUMYAN K, KHACHATRIAN H, MAY J. WARP: Word-level adversarial reprogramming[C]//ACL-IJCNLP 2021-59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Proceedings of the Conference. 2021: 4921-4933.
[26] LIU Z, LIN Y, CAO Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 10012-10022.
[27] LIU Z, LIN Y, CAO Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 10012-10022.
[28] YANG Z, GONG B, WANG L, et al. A Fast and Accurate One-Stage Approach to Visual Grounding[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE Computer Society, 2019: 4682-4692.
[29] LU D, NEVES L, CARVALHO V, et al. Visual Attention Aodel for Name Tagging in Multimodal Social Media[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018: 1990-1999.
[30] LI J, SUN A, HAN J, et al. A Survey on Deep Learning for Named Entity Recognition[J]. IEEE Transactions on Knowledge and Data Engineering, 2022, 34(1): 50-70.
[31] ZHONG Z, CHEN D. A Frustratingly Easy Approach for Entity and Relation Extraction[C]//Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021: 50-61.
[32] YU J, JIANG J, YANG L, et al. Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020: 3342-3352.
[33] ZHANG D, WEI S, LI S, et al. Multi-modal Graph Fusion for Named Entity Recognition with Targeted Visual Guidance[C]//Proceedings of the AAAI conference on artificial intelligence. 2021, 35(16): 14347-14355.
[34] LI L H, YATSKAR M, YIN D, et al. VisualBERT: A Simple and Performant Baseline for Vision and Language [J]. arXiv preprint arXiv:1908.03557, 2019.
[35] HE X, LI S, ZHANG Y, et al. The more quality information the better: Hierarchical generation of multi-evidence alignment and fusion model for multimodal entity and relation extraction[J]. Information Processing & Management, 2025, 62(1): 103875.

Please choose a citation manager

Content to export